(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 

International Bureau 

(43) International Publication Date 
20 November 2003 (20.11.2003) 




PCT 



iiiiiiiiiiiiiiiiiiiiiiiiin 

(10) International Publication Number 

wo 03/096729 Al 



(51) International Patent Classification^: H04Q 7/34 

(21) International Application Number: PCT/IE03/00066 



(22) International Filing Date: 

(25) Filing Language: 

(26) Publication Language: 



8 May 2003 (08.05.2003) 
English 
English 



(30) Priority Data: 

020367 
020674 
020798 



8 May 2002 (08.05.2002) IE 
13 August 2002 (13.08.2002) IE 
4 October 2002 (04. 10.2002) IE 



(71) Applicant (for all designated States except US): ARAN 
COMMUNICATIONS LIMITED [IE/IE]; Blackrock 
Business Park, Carysfort Avenue, Blackrock, County 
Dublin (IE). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): BECK, Philip, 
William [IE/IE]; 18 Domville Grove, Killiney, County 
Dublin (IE). COLLINS, Augustine [IE/IE]; The Haven, 
Walterstown, Kilcullen, County Kildare (IE). MCDON- 
AGH, Brendan [IE/IE]; 67 Maxwell Road, Rathmines, 
Dublin 6 (IE). MORRISROE, Thomas [IE/IE]; Cloon- 
lumney. Church Road, Greystones, County Wicklow (IE). 



(74) Agent: O'BRIEN, John; John A O'Brien & Associates, 
Duncaim House, 3rd floor, 14 Carysfort Avenue, Black- 
rock, County DubHn (lE). 

(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, H, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NI, NO, NZ, OM, PH, PL, PT, RO, RU, SC, SD, 
SE, SG, SK, SL, TJ, TM, TN, TR, TT, TZ, UA, UG, US, 
UZ, VC, VN, YU, ZA, ZM, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZM, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 

European patent (AT, BE, BG, CH, CY, CZ, DE, DK, EE, 
ES, FI, FR, GB, GR, HU, IE, IT, LU, MC, NL, PT, RO, 
SE, SI, SK, TR), OAPI patent (BF, BJ, CF, CG, CI, CM, 
GA, GN, GQ, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— with international search report 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



(54) Title: TELECOMMUNICATIONS NETWORK SUBSCRIBER EXPERIENCE MEASUREMENT 



OS 



102 



Network Entity B 



Packet 
Interface 

103 



Network Entity A 



104 



106 




Alarms 



Reports 



Procedure 
Descriptors 



101 



(57) Abstract: In a mobile network a packet interface (103) is monitored by a tap (160-163) in a non-intrusive manner. Captured 
transaction data is uploaded to a probe (155-158) linked to one or more taps. The probe acts as a slave to a server (159), activating 
and terminating data capture. A coordinator (601) of the probe (155) manages data capture and buffering according to the server. 
The server (150) filters the data according to a subscriber registry (203) and loads data until there are complete protocol descriptions. 
These provide real time subscriber-centered QoS metrics. 
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"Telecommunications Network Subscriber Experience Measurement'* 
INTRODUCTION 
5 Field of the Invention 

The invention relates to customer experience measurement in mobile networks. 
Prior Art Discussion 

10 

Many current mobile communication systems offer a wide range of services, 
including traditional voice telephony, streaming video, email, messaging, and file 
transfer. The consumption of these services by the user places diflFerent demands on 
the capacity of the system. The user's experience of the system thus varies with the 
15 type of service being requested. 

As an example, the end-user requires the system and its services to be highly 
accessible. The system may not be accessible due to the user being out of coverage, 
or due to network equipment being non-operating at some moment in time. Perhaps 

20 the cell is barred, or there is no access to an internal node such as the SGSN in the 
case of GPRS or UMTS. If the system is operating at close to capacity in that area, 
the user may be denied admission for the requested service. In another example, the 
end-user experience is affected by system delays. These may include the delay in 
setting up a connection to the network, the delay in establishing the use of a service, 

25 or the arrival delay of data relating to the service to the mobile station. In a further 
example, the end-user experience is affected by the retainability of the services. These 
may include a voice or data call being dropped rather than surviving until the end- 
user terminates it. A call or connection may persist, but the data service which is 
delivered over that cormection may no longer be in operation. 

30 
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Other examples of end-user QoS experience include variations in delay causing jitter 
and thus malfunction of sti^eaming data services, and reductions in data rate due to 
congestion in the mobile system. 

5 Currentiy, network operators have limited knowledge of QoS as experienced by the 
end-user. This is especially so in emerging mobile networks offering advanced data 
services, as it is diflBcult to monitor OoS from measurements taken in die network 
infrastructure alone. One approach to monitoring QoS is so-called drive-testing, in 
which a specially equipped mobile station is brought to a predetermined location in 

10 the network, measures air interface parameters at that location, and uploads them to 
an analysis system at a later stage. A limitation of such an approach is that it collects 
information about the air interface at a specific location only, and iaformation only 
about a pre-determined service usage. 

15 It is also know to analyse some user-specific criteria to assess adherence to service 
level agreements (SLAs). Such SLAs are sometimes part of the FCAPS (Fault, 
Configuration, Accoimting, Performance, and Security) procedure devised by the 
Telemanagement Forum ("TMF''). This is performed by offline analysis of call 
detail records ("CDRs'O generated by some network elements for billing purposes. 

20 While this approach does provide some user-specific data it is of a limited extent and 
is effectively historical. 

The invention is therefore directed towards providing for improved customer-centric 
quality-of-service measurement. 

25 

SUMMARY OF THE INVENTION 

According to the invention, there is provided a management system for a 
telecommunication network, the management system comprising: 



30 
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a plurality of taps for monitoring in real time activity at network interfaces to 
provide network transaction data, 

a filter for jBltering the transaction data on a subscriber identifier basis to 
5 provide per-user quality-of-service metrics in real time, and 

a server for aggregating the metrics and for storing at least some of the metrics 
and transaction data. 

10 By tapping activity at interfaces and filtering on a subscriber basis the invention 
provides comprehensive subscriber-centric quality-of-service data. 

In one embodiment, the taps each capture network messages and generate a 
transaction event record for one or more messages, the transaction records being 
15 provided as the transaction data. 

In another embodiment, at least some of the taps are non-invasive. 

In a ftirther embodiment, at least one tap is connected to monitor traffic at a network 
20 interface between network elements handling messages for multiple subscribers. 

In one embodiment, at least one tap is a software agent executing on a subscriber 
device. 

25 In another embodiment, the software agent executes on a SIM card of a subscriber 
mobile device or in the device's circuit. 

In a further embodiment, there are a plurality of probes located remotely from the 
server and connected to at least one tap. 

30 
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In one embodiment, each probe comprises a task coordinator, and a registry storing 
data concerning taps presently linked to the probe and characteristics of the network 
interfaces where the taps are located. 

5 In another embodiment, the registry holds data concerning mobile terminal 
configurations. 

In a further embodiment, the task coordinator manages commands from the server 
defining monitoring tasks including start times and end times. 

10 

In one embodiment, the task coordinator comprises a function for receiving filtering 
commands from the server for filtering in the probes. 

In another embodiment, the probe buffers transaction data for periodic upload to the 
15 server. 

In a further embodiment, the server polls the probes for transaction data uploads. 

In one embodiment, the server comprises a collection process associated with each 
20 tap or each probe, at least some collection processes performing format conversion or 
decryption on data received from an associated probe. 

In another embodiment, the server comprises a subscriber registry for storing 
identifiers of subscribers for whom quality-of-service metrics are to be determined 
25 and for transferring identifiers to the filter, located either in the server or in a probe. 

In a fixrther embodiment, the server comprises a data loading ftmction for loading 
filtered data into memory. 

30 In one embodiment, the data loading ftmction performs transaction data. 
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In another embodiment, the server comprises a filtering memory structure for 
incomplete records and the data loading function writes filtered data to said memory 
structure, and monitors the data to determine when a complete protocol procedure 
5 description for a subscriber has been loaded, and transfers the complete descriptors as 
metrics to a complete records memory structure. 

In a further embodiment, the server comprises a report generating function for 
analysing the metrics and generating reports according to operator configurations. 

10 

In one embodiment, the server comprises an alarm generating function for analysing 
the metrics and generating alarms according to operator configurations. 

In another embodiment, the configurations comprise service level agreements. 

15 

In a further embodiment, the server comprises a publish-and-subscribe mechanism to 
allow remote mechanisms to receive alarm notifications. 

In one embodiment, the thresholds are set by Key Performance Indicators. 

20 

In another embodiment, the metrics include: 
attach success rate, 

time to complete an Attach procedure, 

detach success rate, 
25 time to complete Detach procedure, 

abnormal termination rate and cause, 

PDF Context Activation success rate, 

time to complete PDP Context Activation procedure, 

PDP Context De-activation success rate, 
30 time to complete PDP Context De-activation procedure. 
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PDP Context Abnormal De-activation rate & cause, or 
PDP Context Throughput in uplink & downlink. 

In a further embodiment, the metrics include:- 
service activation success rate, 
service completion success rate, 
service average bitrate, 
service startup and shutdown latencies, 
how often the actual bitrate is within x% of maximum bitrate, 
how often SDUs are delivered out of order, 
number of SDUs lost or detected as erroneous, 
residual bit error rates in SDU's subflows, 

how often the transfer delay of SDUs is within x% of the maximum allowed, 
or 

how often the actual bitrate is within x% of guaranteed bitrate. 

In one embodiment, the metrics are classified in one or more of:- 

a conversation class of telephony speech, VoIP, audio conferencing, or VPN, 

a streaming class of one-way video streams (e.g. sports highlights, music 
videos, security camera feeds), or one-way audio streams (e.g. music or sound 
broadcasts), 

an interactive class of database retrieval, client/server interactions, browsing 
and Intemet access, WAP access, process control, remote sensing, remote 
control, or file transfer, and 

a background class : non-urgent measurement collection, email, or 
SMS/MMS. 



DETAILED DESCRIPTION OF THE INVENTION 
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Brief Description of the Drawings 

The invention will be more clearly understood from the following description of 
some embodiments thereof, given by way of example only with reference to the 
5 accompanying drawings in which:- 

Fig. 1 is a flow diagram of the data flows within a mobile telecommunications 
network management system of the present invention; 

10 Fig. 2 is a block diagram illustrating architecture of the management system; 

Fig. 3 depicts internal functional components of a server of the management 
system; 

15 Fig. 4 depicts message flow between entities of the management system; 

Fig. 5 depicts an example of protocol events being sent from a mobile station 
to the network, such events being examples of traffic monitored in real time 
by the management system; 

20 

Fig. 6 depicts internal fixnctional components of the management system 
which manage agents running on mobile terminals and produce data for the 
management system; and 

25 Fig. 7 shows an example of task coordination messages and data being 

exchanged between a task coordinator and agents running on mobile 
terminals. 



Description of the Embodiments 

30 



wo 03/096729 



PCT/IE03/00066 



Referring to Fig. 1, a network entity A 101 is interfaced to a network entity B 102 by 
means of a network packet interface 103. This interface 103 carries packet traffic 
including packets containing data about the behaviour of a high level protocol such 
as GPRS. In the example of GPRS and UMTS, the interfaces are identified in the 
5 relevant 3GPP and ETSI standards as the Gb (GPRS) or lu (UTRAN), Gn, and Gi 
interfaces. The packets are captured by taps at the interfaces and are processed in a 
step 104 by probes. This filtering eUminates packets which do not carry Ihe required 
information about the protocol events or services. 

10 A protocol such as GPRS or UMTS contains protocol events relating to procedures 
such as, for example, attaching to the network, detaching from the network, and 
activating and de-activating a PDP context. Similarly, service information which is 
useful in determining the QoS being experienced by the user includes URLs being 
visited, the behaviour of email using POP3, FTP behaviour, video and audio 

15 streaming, and X.25 or IP data services. The next step, 105 involves using a 
subscriber id to retain only those protocol events and service usage data relating to 
such specified subscribers. In the case of GPRS and UMTS, the subscriber id known 
as IMSI is used to discriminate subscribers. This step also involves aggregating 
together protocol events which contribute to a more complete protocol procedure so 

20 as to produce a single procedure descriptor for each procedure executed by each 
subscriber. For example, in the case of GPRS, activating a PDP context involves 
several protocol events, such as PDP context activate request, PDP context 
activation accept, 2ind PDP context activation complete. 

25 By the time filtering 104 and aggregation 105 are complete, a set of discrete protocol 
procedure descriptors have been produced ft-om a series of packets on the interface. 
These descriptors provide metrics including Key Performance Indicators which have 
been defined on the protocol. Such KPIs can include for example latencies, delays, 
success rates, and throughput values. These descriptors are stored and processed by 
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a server in step 106, and from them alarms may be raised and reports may be 
produced. 

A sei-ver cormected to the probes carries out the steps 105 and 106. The activity of 
5 step 104 is carried out by a probe connected to the network tap. 

Referring to Fig. 2 a management system to implement the steps 104-106 comprises: 
a server 150; 
probes 155-158; and 
10 network interface taps 160-163. 

The network comprises a mobile station 160, a BSS 171, an SGSN 172, a GGSN 
173, and network services 174. Probe B 156 is connected to the tap 161 attached to 
the Gb interface (GRPS) or lu interface (UMTS), and provides the following 
15 protocol procedures and data fields:- 

• ATTACH : IMSI, cell-id, timestamp, procedure duration, result 

• DETACH : IMSI, cell-id, timestamp, procedure duration, result 

• PDP CONTEXT ACTIVATION : IMSI, ceU-id, timestamp, procedure 
duration, result, origin, data address, QoS Negotiated, QoS Requested 

20 • PDP CONTEXT DEACTIVATION : IMSI, cell-id, timestamp, procediire 
duration, result, origin, number of bytes sent & received. 

The management system monitors activity at the various interfaces to gather 
transaction data and to filter this data to provide subscriber-centric QoS metrics in 

25 real time. The taps 161-163 are non-invasive insofar as they do not impose an 
overhead on network elements or affect trajBfic across the interfaces. The tap 160 is 
an agent executing on the mobile station, and may thus impose a minimal overhead. 
Because some of the filtering operations are provided by the distributed probes, they 
can be very quickly performed in a dynamic real time manner. The filtered data 

30 delivered to the server 150 can be used and stored in a variety of ways as desired by 
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the network operator. The probes 155-158 temporarily store the filtered transaction 
data in memory files which are frequently uploaded by collection processes in the 
server 150. This ft-equency may be as high or low as required by the operator. The 
server had ftmctions for various activities including alarm detection for real time or 
5 near real time generation of alarm events on a subscriber-centric basis. 

The link between the probes 155-158 and the taps 160-163 is very specific to the 
nature of the interface concerned and the construction of the taps and probes. For 
example, the links are different for El 2Mb/s GPRS and LAN 100 Mb/s network 
10 interfaces. Also, even for one type of network interface such as El 2Mb/s, the 
different taps may handle data differently. However, the links between the probes 
155-158 and the server are uniform, there being one interface protocol for each 
network interface. The low-level interfacing functions are below the interrupted lines 
within the probes 155-158. 

15 

Referring to Fig. 3, this depicts the internal ftxnctional components of the server 150. 
The interface between the probes 155-158 and the server 150 has certain 
characteristics. An unambiguous definition of the subscriber-related data (and other 
data as appropriate) which is provided by the probes 155-158 is required. This data 

20 may be produced periodically or in real-time (or near real-time). This data is 
transferred in a defined manner to the server 150. The server 150 has the ability to 
control and manage the production of data by the probes 155-158. All of these 
characteristics foUow a proprietary scheme, or altematively may be based on an 
existing standard such as for example the 3GPP Performance Management 

25 Integration Reference Point, as described in the document 3GPP TS 32.401 
"Performance Management, Concepts & Requirements". 

A collection process 202 in the server 150 fetches the data periodically ft'om an 
associated probe once it is produced. This involves acquiring the data from a probe 
30 155-158, for example by means of FTP, and as necessary decrypting and deciphering 
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the format of the incoming data. Different collection processes 202 cater for diflFerent 
types of probe and their interfaces. Multiple data sources are supported, dependent 
on time synchronisation and correlation requirements. 

5 Once the data has been fetched by the server 150, it is filtered by subscriber identifier. 
A subscriber registry 203 contains the subscriber ids (for example IMSI) which are of 
interest to the operator. There is a subsystem which manages the contents of the 
subscriber registry 203 and allows properly authorised users to inspect, alter, add, 
and delete subscriber ids in the subscriber registry 203. Filtering occurs in the server 

10 as part of data loading by a function 204, In another embodiment, a set of subscriber 
ids may be sent to the probe in order to configure it to cany out the filtering in a 
distributed manner. 

The data loading function 204 extracts the fields of interest from each protocol event 
15 which is allowed through the filter, and stores these in an incomplete records table 
206. As each protocol event which goes to make up a procedure arrives, the fields of 
interest are extracted and stored. At a certain point, different for each procedure, all 
the fields of interest for the procedure are obtained, and a completed protocol 
procedure descriptor is stored in a complete records table 207. An example of the 
20 fields of interest from the iucoming protocol events for the GPRS procedure PDP 
Context Activation includes fields for the timestamp of the PDP Context Activation 
Request, and the timestamp for the PDP Context Activation Complete. From this, 
the data loading function 204 can fill in the duration field in tiie procedure descriptor 
in the complete records table 207 for the PDP Context Activation procedure. As an 
25 example of a possible KPI based on this data, the operator may be interested ia the 
mean time to set up a PDP Context for aU premium users. This example can be 
extended to provide KPIs dealing with other important aspects of service offered to 
groups of users, such as attach latencies, throughput, error rates, service denial rates, 
abnormal termination rates, and comparisons between QoS levels requested. 



30 
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A report generation function 208 generates reports per subscriber, per cell, or per 
service (or per APN if appropriate). These reports may refer to different periods of 
time, such as daily, weekly, or monthly. The reports capture the user experience of 
eitlier individual users or user groups. Users may be ranked on criteria such as 
5 throughput or perceived quality. Violations of QoS thresholds specified in KPIs 
which form part of SLAs are included in reports. The reports are dej&ned and set up 
in a report management function 212 and are made available to a client system 213. 

An alairm generation function 209 generates alarms when KPI values breach 
10 predefined thresholds. This may be used to monitor adherence to Service Level 
Agreements (SLAs). These alarms £ire made available outside the server 150 by 
means of a publish-subscribe mechanism - consumers of the alarms subscribe via the 
alarm subscription subsystem 210. Those skilled in the art wiU recognise that alarms 
may, for example, be made available by means of the 3GPP standard Corba IRP for 
15 alarms, by means of a customised interface, by email, text, or TCP/IP socket. 
Alarms may be delivered to the user by means of the client system 213 or to an 
extemal alarm subscriber 214. Remote client systems may received alarms and 
notifications on the basis of a pubhsh-and-subscribe mechanism of the server. 

20 A collection management function 211 is responsible for managing the data 
collection and storage. It sets up the parameters for collecting the data, and associates 
a particular collection process 202 vnth a probe 155-158. It sets up the parameters for 
a data ageing and aggregation function 205, including the time period for which raw 
data is held before it is aggregated or summarised for longer-term storage. 

25 

Referring to Fig. 4, a typical sequence of messages exchanged between the server 150 
and the probe 155 is shown. When the server 150 requires a packet monitoring task 
to be started, a start message 303 is sent to the probe 155. This message includes a 
command to start monitoring, as well as other configuration information which may 
30 include a unique monitoring task id, a description of which packet events are to be 
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collected, and a granularity period defining how often data items are to be packaged 
and sent to the server 150. When the probe 155 receives this message, it carries out 
initialisation activities in preparation for sending periodic results 304 to the server 
150. 

5 

At some time after a granularity period has completed, the data items for that period 
are packaged and sent to the server 150 in a periodic results message 304 by the 
probe 155. This is repeated for each granularity period. The periodic results are 
received by the server 150, and stored and processed as described above. The 
10 periodic results message 304 may contain addressing information dejBning the sender 
and receiver of the message, the unique monitoring task id, the data items, and 
appropriate status information. 

At some future point, the server 150 sends a stop 305 message to the probe 155. This 
15 causes the probe 155 to carry out varioxxs termination activities which leave the probe 
155 in an appropriate state for fiirther monitoring tasks to be started at some later 
stage. 

The system has built-in safeguards to deal with unexpected situations. As an 
20 example, if a new start message 303 is received by the probe 155 before the currently 
executing monitoring task is stopped by means of a stop 305, the probe will make its 
best effort to satisfy the requirements of both tasks for the period during which they 
are both executing. In a further example, if the start 303 message requests specific 
data to be collected which the probe is not capable of collecting, for example because 
25 the specific data is not available on the interface, then this will be notified to the 
management system 301 by means of the status information in the periodic results, 
or by some other appropriate means. 

In a further example, the operator may be concerned about security issues, and may 
30 require a correct response to an authorisation challenge from the probe 157 to the 
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server 150 to ensure that subscriber usage pattern information is only sent to 
authorised consumers. This could be implemented for example by includmg a 
password in the start message, or by an additional protocol step to challenge the 
server 150. Further security could be applied by requiring the periodic results to be 
5 sent over a well-established secure cotmection methodology such as, for example, 
IPSEC or SSL. 

Referring again to Fig. 2, examples of Key Performance Indicators based on the 
captured information from the probe 156 could include :- 
10 • Attach Success Rate : (number of ATTACHes where ATTACH result 

code=successful) / (total number of ATTACHes.) 

• Abnormal Termination Rate : (number of PDF CONTEXT 
DEACTIVATIONS where result code <> OK) / (total number of PDF 
CONTEXT DEACTIVATIONS). 

15 • Average PDF context throughput in uplink : (PDF CONTEXT 

DEACTIVATION number of bytes sent) / (PDF CONTEXT 
DEACTISTATION timestamp - PDF CONTEXT ACTIVATION 
timestamp). 

• Protocol latency leading to delay before usefixl data begins to flow : PDF 
20 CONTEXT ACTIVATION procedure duration. 

In another example of the application of the invention to GPRS and UMTS, probe C 
157 is attached to the Gn interface. This interface carries unciphered data packets to 
and from the network services. Probe C 157 is capable of examining these packets on 
25 the fly and producing data about the services being consumed by users. Examples of 
key performance indicators available by probing this interface include (and are not 
limited to) :- 

- Service activation success rate 

- Service completion success rate 
30 - Service throughput 
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- Service usage profUes 

- Service SLA parameter adherence 

- Service startup latency 

- Service shutdown latency 

5 

In another example, probe A 155 monitors at the mobile terminal user-visible QoS 
metrics such as service latencies and service success rates. Probe A 155 may in one 
embodiment execute on the mobile station, for example in the SIM card, or may be a 
separate entity in communication with the mobile terminal. 

10 

In another example, the probe C 157 also monitors Gn traflfic between GSNs 
(including those interconnected via a GRX equipment) which provides metrics on 
roaming and mobility management topics. 

15 In another example, the probe D 157 monitors the Gi reference point which may be 
comprised of several types of interfaces, including IP and X.25, and probe D 158 
provides metrics on these interfaces including interface usage profiles and interface 
latencies. 

20 In another example, market survey information 159 may be loaded into the server 
150. In a tj^ical mobile operator, the marketing function interviews selected user 
groups to determine what their subjective experience of the network is. The present 
invention allows user experience to be measured from the equipment in a weU- 
defined and objective manner. It is of major benefit to the operator to be in a position 

25 to compare the user experience as measured by the equipment and as collected by the 
market surveys. It is imperative that the market surveys are designed carefully and 
the set of measurements and the calculations performed on them are chosen 
carefully, so that comparable measurements are being made. As an example, a 
weighted average of a set of equipment measurements may be calculated to reflect 

30 the subjective importance of different measurements when assessing the user's 
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experience of the network. The results of this weighted average may be compared 
with the survey results by means of an accepted statistical technique (for example, a 
bar chart showing the two sets of results side by side). 

5 Fig. 5 shows an illustrative example of protocol events being sent between a mobile 
station and a network. These events are monitored by the tap 160, which generates 
one transaction event per operation. For example, the first three messages GPRS 
ATTACH REQUEST, GPRS ATTACH ACCEPT, AND GPRS ATTACH 
COMPLETE are configured to complete one transaction event. One event is 
10 uploaded by the tap 160 to the probe 155. The transaction events are filtered by the 
probe 155 to provide per-user events, as configured by the server 150. 

Tuming to Fig. 6, operation of the system to monitor aspects of mobile terminal 
behaviour which give further insight into the subscriber's experience of the mobile 

15 system as illustrated. This figure depicts the internal functional components of the 
probe 155 and an agent 160 for monitoring mobile terminal behaviour in accordance 
with the present invention. In this scenario the item 160, while referred to generally 
as a "tap'* in the context of the items 160-163 is more correctly referred to as a 
software agent. This is because it executes on a mobile device 170 SIM on its circuit 

20 processor or in its SIM card. On the other hand the items 161-163 are taps at 
network interfaces. Where the agent executes on the device's circuit itself, it may be 
downloaded as an applet. 

The probe A 155 comprises a task coordinator 601 and a registry 602. The registry 
25 602 holds details of which mobile terminals have an agent 160 installed on them, and 
any configuration or variant information required about each agent. The registry 602 
also holds profiles defining which data counters are available on mobile terminal 
types, as these vary gready from model to model and between mobile terminal 
software installation levels. Each mobile terminal has a unique identifier in the 
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context of the registry, which is used to distinguish commands to the mobile terminal 
and data retuming from it. 

The task co-ordinator 601 receives a request from the server 150 which defines a 
5 monitoring task on one or a pluraUty of mobile terminals 170. This request also 
specifies which data and events are to be collected as this monitoring task proceeds. 
A start-time and end-time may also be specified in the request. Appropriate 
commands are sent from the task co-ordinator 601 to the agent 160 to cause 
monitoring to commence or stop as required. 

10 

As data is produced periodically by each mobile terminal 170, it is transimtted via 
the available data transfer mechanisms of the intervening mobile network to the task 
co-ordinator 601. The task co-ordinator 601 processes these periodic data 
transmissions, aggregating data from a plurality of mobile terminals, and mediating 
15 the data into a consistent format and presentation. It is important to avoid 
overloading the network with data transmissions - hence, mechanisms wiU be in 
place to reduce or avoid traffic during busy periods, and to smooth out peaks in 
trajfific, and to compress data before transmission. 

20 This data is then made available to the server 150 in a similar manner to the other 
probe types illustrated in Fig. 2, and the server 150 stores, processes, analyses, and 
visuaUses the data in a manner which is usefiil to the network operator in assessing 
the end-user's experience of the network's QoS. 

25 Fig. 6 also depicts the intemal architecture at the mobile terminal 170. The agent 
160 may either be pre-instaUed before the mobile terminal 170 is sent to the field, or 
is installed over-the-air by an industry-standard method. 

The monitoring task description received by the agent 160 from the task co-ordinator 
30 601 includes a description of the data to be collected and the granularity of the 
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collection. The agent 160 interacts with the SIM card 603 and/or the mobile 
equipment 604 to collect this information as required. The data available depends on 
the mobile equipment type and the features which are supported by the software 
variant on the mobile equipment. This is defined, as stated above, in the profile of 
5 the mobile ternainal type in the registry 602. As an example, the agent will report 
configuration and status information. For example, the agent may run an AT 
command to retrieve the actual manufacturer model, revision, and serial number of 
the mobile equipment - this information may be used to cross-check that the correct 
profile ra the registry 602 is being used. In another example, the agent 160 may read 
10 the current battery level, which wiU affect whether the monitoring task can 
successfully execute throughout its defined activity period witiiout the battery 
running out. 

Tuming to Fig. 7, a typical sequence of messages exchanged between the task 
15 coordinator 601 and the agent 160 running on the mobile terminal 170 is shown. 
When the task coordinator 601 requires a monitoring task to be started, a Start 
Monitoring Task message 703 is sent to agent on the appropriate mobile terminal. 
This message includes addressing information defining the sender and receiver of the 
message, a command to start monitoring, a unique monitoring task id, a description 
20 of which data items are to be collected, and a granularity period defining how often 
data items are to be packaged and sent to the task coordinator 601. When the agent 
606 receives this message, it carries out various initialisation activities in preparation 
for sending periodic results 404 to the task coordinator 601. 

25 At some time after a granularity period has completed, the data items for that period 

are packaged and sent to the task coordinator 601 in a Periodic Results message 704 
by the agent 606. This is repeated for each granularity period. The periodic results 
are received by the task coordinator 601, stored, processed, analysed and visualised 
as previously described. A Periodic Results message 704 contains addressing 
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information defining the sender and receiver of the message, the unique monitoring 
task id, the data items, and appropriate status information. 

When the finish time of the monitoring task has passed, the task coordinator 601 
5 sends a Stop Monitoring Task 705 message to the agent 160, This causes the agent 
160 to carry out various termination activities which leave the agent 166 and the 
mobile terminal 170 in an appropriate state for farther monitoring tasks to be started. 

The invention has built-in safeguards to deal with unexpected situations. As an 
10 example, if a new Start Monitoring Task message 703 is received by the agent 160 
before the currently executing monitoring task is stopped by means of a Stop 
Monitoring Task 705, the agent will make its best effort to satisfy the requirements of 
both tasks for the period during which they are both executing. In a further example, 
if the Start Monitoring Task 703 message requests specific data to be collected which 
15 the agent is not capable of collecting, for example because there is no programmed 
interface on the mobile terminal in question to support the retrieval of the specific 
data, then this wiU be notified to the server 150 means of the status information in 
the Periodic Results, or by some other appropriate means. 

20 In a further example, the subscriber may be concemed about security issues, and 
unauthorised access to the subscriber's SIM card and information about the 
subscriber's usage of the network. A password may be placed in the registry 602 and 
also in the agent 160 at the moment it is installed in the SIM. When the task 
coordinator 601 sends the Start Monitoring Task 703 message, it can optionally 

25 include the password retrieved fiom the registry for this subscriber. The agent 160 
will then compare the password in the message with the password it has stored 
within itself, and allow the monitoring task to proceed only if there is a match. 

Those skilled in the art will recognise that the mechanisms for transmitting data 
30 through the network and over the air to the agent 606 and for retrieving SIM data 
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could use SMS messages. Further possibilities include sending data messages via a 
PDP context or some other connection-oriented or connection-less data transfer 
mechanism. The invention specifies the process of monitoring the end-user's 
experience of service quality, independent of the mechanism used for communicating 
the monitored information across the network. 

Some example use cases and scenarios describing how the network operator may 
benefit from the invention are as follows. 

(a) An operator offers Gold Service to all users belonging to a single corporate 
customer (for example bank). The operator wishes to implement a Service 
Level Management capability which monitors the service level supplied to all 
Gold Class users, comparing the service level against the level specified in the 
Service Level Agreement for such users, and taking appropriate action if the 
level is not sufficient. 

(b) An operator wishes to provide proactive Customer Care. The management 
system detects poor or deteriorating user experience of the network, possibly 
before the users themselves realise it. An alarm is generated which is subscribed 
to by the Customer Care system which has the opportunity to deal with it 
before the user reports the situation. This mi^t involve contacting the user 
with an assurance that improvements are being made, or with some 
recompense, for example. 

(c) An operator wishes to carry out service impact estimation. By inspecting the 
previous effect on user experience when a new service is introduced, estimates 
can be produced about the effect of introducing a new service to the customer 
base. 
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(d) An operator wishes to check whether a particular user equipment type delivers 
good QoS or not. This naay involve for example correlating service usage 
patterns with handset type, or correlating QoS issue occurrence with handset 
type. This may be of particular interest to an operator in monitoring the 
5 performance of new handset types when they are launched. 



(e) A super-operator wishes to check that a roaming user gets the same QoS in 
each network belonging to the super-operator. 

10 (f) An operator traces QoS as user roams in-caU between cells. The operator 
wishes to monitor how roaming users use the network, and proactively react to 
problems experienced by this class of user. The operator may wish to focus on 
QoS improvements in order to retain this class of floating customer, as they 
may generate high revenues. 

15 

(g) An operator checks whether the services most consumed by a Gold Class user 
are those listed in the user's service definition. 

(h) An operator compares bad user experience as detected by the management 
20 system (e.g. in a certain cell accessing certain services, or while roaming) with 

KPI breaches obtained from NMS / EM. 



The overall intent of these examples is to show what the benefits to the operator of 
the management system would be. These include understanding the coimection 

25 between service usage and quality, understanding what type of site produces the 
most revenue, proactively managing the response to SLA violations by means of for 
example automatic discounting, ensuring the successful launch of a new service, 
understanding what effect poor performance has on customer usage of a service, 
detecting negative or poor customer experience, optimising the customer experience, 

30 and correlating network statistics with customer experience. 
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It will be appreciated that the management system converts protocol events into 
network-based QoS metrics which will cater for large networks and are available in 
near real-time. It also converts service usage data into user-based QoS metrics which 
5 will cater for large numbers of users and cire available in near real-time. The 
management system also converts service usage data into network-based QoS 
metrics which will cater for large networks and are available in near real-time. The 
management system also allows the definition of Key Performance Indicators (KPIs) 
with associated thresholds which are based on end-user experience of the network. 
10 This allows the operator to support and manage the definition of Service Level 
Agreements (SLAs) related to the end-user's experience of the network. 

In the case of WAP access via GPRS, relevant measures include per user statistics, 
correlations between page traffic and page performance, throughputs, page size, 

15 pages visited, dovmload times, page availability, errors, data transfer efficiency 
(payload/total packet size), and abandoned, failed, and successful hits. In the case of 
Intemet access via GPRS, relevant measures include per user statistics, correlations 
between page traffic and page performance, throughputs, page size, pages visited, 
download times, page availability, errors, data transfer efficiency (payload/total 

20 packet size), and abandoned, failed, and successful hits. In the case of VPN via 
GPRS, relevant measures include per APN statistics, GTP tunnel creation success 
rates, abandoned, failed, and successful VPN element setups and terminations. In 
the case of MMS via GPRS, relevant measures include breakup between WAP and 
SMTP delivery, per-APN statistics, time to delivery, average/actual message size, 

25 request/ delivery success rates, throughputs, payload types. 

The invention is not limited to the embodiments described but may be varied in 
construction and detail. For example, the management system may be used by a 
land-line teleconamunication network operator. Also, the data may be pushed by the 
30 probes to the server, rather than being transmitted in response to a polling signal. 
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Claims 

1. A management system for a telecommunication network, the management 
system comprising: 

5 

a plmality of taps for monitoring in real time activity at network interfaces to 
provide network transaction data, 

a filter for filtering the transaction data on a subscriber identifier basis to 
10 provide per-user quality-of-service metrics in real time, and 

a server for aggregating the metrics and for storing at least some of the metrics 
and transaction data. 

15 2. A management system as claimed in claim 1, wherein the taps each capture 
network messages and generate a transaction event record for one or more 
messages, the transaction records being provided as the transaction data. 

3. A m€inagement system as claimed in claims 1 or 2, wherein at least some of 
20 the taps are non-invasive. 

4. A management system as claimed in any preceding claim, wherein at least 
one tap is connected to monitor traffic at a network interface between 
network elements handling messages for multiple subscribers. 

25 

5. A management system as claimed in any preceding claim, wherein at least 
one tap is a software agent executing on a subscriber device. 

6. A management system as claimed in claim 5, wherein the software agent 
30 executes on a SIM card of a subscriber mobile device or in the device's circuit. 
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7. A management system as claimed in any preceding claim, wherein there are a 
plurality of probes (155-158) located remotely from the server and connected 
to at least one tap (160-163). 

5 

8. A management system as claimed in claim 7, wherein each probe (155-158) 
comprises a task coordinator (601), and a registry (602) storing data 
concerning taps presently linked to the probe and characteristics of the 
network interfaces where the taps are located. 

10 

9. A management system as claimed in claim 8, when dependent on claims 5 to 
7, wherein the registry holds data concerning mobile terminal configurations. 

10. A management system as claimed in claim 8 or 9, wherein the task 
15 coordinator (601) manages commands from the server (150) defining 

monitoring tasks including start times and end times. 

11. A management system as claimed in claim 10, wherein the task coordinator 
comprises a ftmction for receiving filtering commands from the server for 

20 filtering in the probes. 

12. A management system as claimed in any of claims 5 to 11, wherein the probe 
buffers transaction data for periodic upload to the server (150). 

25 13. A management system as claimed in claim 12, wherein the server (150) poUs 
the probes for transaction data uploads. 



14. 



A management system as claimed in any preceding claim, wherein the server 
(150) comprises a collection process (202) associated with each tap (160-163) 
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or each probe (155-158), at least some collection processes performing format 
conversion or decryption on data received from an associated probe. 

15. A management system as claimed in any preceding claim, wherein the server 
5 comprises a subscriber registry for storing identifiers of subscribers for whom 

quality-of-service metrics are to be determined and for transferring identifiers 
to the filter, located either in the server or in a probe. 

16. A management system as claimed in any preceding claim, wherein the server 
10 comprises a data loading ftmction (204) for loading filtered data into memory. 

17. A management system as claimed in claim 16, wherein the data loading 
fixnction performs transaction data. 

15 18. A management system as claimed in claims 16 or 17, wherein the server 
comprises a filtering memory structure (206) for incomplete records and the 
data loading ftmction writes filtered data to said memory structure, and 
monitors the data to determine when a complete protocol procedure 
description for a subscriber has been loaded, and transfers the complete 

20 descriptors as metrics to a complete records memory structure. 

19. A management system as claimed in any preceding claim, wherein the server 
comprises a report generating ftmction (208) for analysing the metrics and 
generating reports according to operator configurations. 

25 

20. A management system as claimed in any preceding claim, wherein the server 
comprises an alarm generating ftmction (209) for analysing the metrics and 
generating alarms according to operator configurations. 
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21. A management system as claimed in claim 20, wherein the configurations 
comprise service level agreements. 

22. A management system as claimed in claims 19 to 21, wherein the server 
comprises a publish-and-subscribe mechanism to allow remote mechanisms to 
receive alarm notifications. 



23. A management system as claimed in claims 19 to 22, wherein the thresholds 
are set by Key Performance Indicators. 

10 

24. A system as claimed in any preceding claim, wherein the metrics include: 

attach success rate, 

time to complete an Attach procedure, 

detach success rate, 
15 time to complete Detach procedure, 

abnormal termination rate and cause, 

PDP Context Activation success rate, 

time to complete PDP Context Activation procedure, 

PDP Context De-activation success rate, 
20 time to complete PDP Context De-activation procedure, 

PDP Context Abnormal De-activation rate & cause, or 

PDP Context Throughput in uplink & downlink. 

25. A system as claimed in any preceding claim, wherein the metrics include:- 
25 service activation success rate, 

service completion success rate, 

service average bitrate, 

service startup and shutdown latencies, 

how often the actual bitrate is within x% of maximum bitrate, 
30 how often SDUs are deUvered out of order, 
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number of SDUs lost or detected as erroneous, 
residual bit error rates in SDU's subflows, 

how often the transfer delay of SDUs is within x% of the maximum 
allowed, or 

how often the actual bitrate is within x% of guaranteed bitrate. 

26. A system as claimed in any preceding claim, wherein the metrics are classified 
in one or more of:- 

a conversation class of telephony speech, VoIP, audio conferencing, or 
VPN, 

a streaming class of one-way video streams (e.g. sports highlights, music 
videos, security camera feeds), or one-way audio streams (e.g. music or 
sound broadcasts), 

an interactive class of database retrieval, client/server interactions, 
browsing and Internet access, WAP access, process control, remote 
sensing, remote control, or file transfer, and 

a background class : non-urgent measurement collection, email, or 
SMS/MMS. 

27. A management system substantially as described with reference to the 
drawings. 
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